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Abstract 

The relationship between reputatlonal ratings of graduate 
chemistry, history, and psychology departments and their faculties 
scholarly productivity was examined using data from a national 
sample of departments in each field. Six measures of productivity, 
three for the entire career, and three for the most recent three 
years, were related to ratings of the "quality" of the faculty, \ 
which were obtained by following procedures used by Roose and 
Andersen (1970). Although some statistically significant relation- 
ships were obtained, a close examination, of the data Indicated that 
ratings are Inadequate assessments of the scholarly contribution of 
faculty or the "quality" of departments. 



^ .Relationship Between Ratings of Graduate Departments 
and Faculty Publication Rates 



A 



There has been a long controversy about the relationship - 
between a department's scholarly productivity and its reputation 
within its field. Some researchers have reported a substantial 
positive relationship— i.e., the more the faculty in a department 
publish, the higher its standing with other faculty in the field- 
thus promoting the idea that these are mutually supportive Indicators 
of the scholarly contribution and "quality" of the department. 
' Certainly scholarly contributions have been of primary importance in 
the evaluation of research, graduate training, and quality of 
faculty in schools and departments in universities. Publications 
are evaluated to provide evidence for quality of scholarship both 
for individuals and departments (see Jones, 1980, for an excellent 
review of the issues involved in assessing scholarship through 
publications) , A number of studies have found substantial correla- 
tions between publication activity and other indicators of scholarship 
(e.g., Clark, Hartnett, & Baird, 1976; Hagstrom, 1971). However, 
some have argued that publication rates are Inadequate as measures 
of scholarship, and that undue emphasis on sheer high publication 
rates is ultimately damaging to quality. More importantly, critics 
argue that high publication rates are unrelated or even negatively 
related to quality of teaching arid overall contribution to the 
discipline through professional activities. That is, a faculty 
member may be an excellent professor and member of the profession 
without having a high output of articles, and a department may be an 
This study was supported by the Graduate Record Examinations Board. 
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excellent place to work and study without haying a-Agh average 
publication rate.. Conversely, a faculty member ^ er 
pressure to ''publish or perish" .may neglect his or her teach£i& 
duties and professional commitments to. dShcin'ttate p^^snlng. 
• Although publication rates have-been widely c^ticize^as ^yJr 

■ * - 5 ■ ■' • 

indicators "of departmental quality, /there has been even gr^te/ ■ \ 
controversy over the meaning and utility of reputational ratings .. ; 
of the "quality" of graduate departments (see Clark, 1976, for a 
summary of the issues) . Doubts have been expressed about their 
significance for different kinds of programs, their accuracy, ^ . 
their relation to graduate students' learning experiences, etc. 
However, the high degree of relationship between' scholarly produc- 
tivity and.reputational ratings have been used to def-d the use 
of each as indicators of' program quality; i.e., the high relationship 
with, publication rates has been used to suggest that reputational 
ratings were valid, and the high relationship with reputational 
ratings to show that publication rates are a legitimate measure 
of quality. Thus the degree of relationship between these two 
variables becomes critically important. Are they really highly 
related, and can they be used as indicators of overall "quality"? 

Earlier studies had found substantial relationships between 
the variables but had not controlled, for size; thus the larger and 
usually more prestigious departments tended to appear more productive, 
in contrast, in a recent article and a series of comments in the 
togricaa Psychologist centering on the relationship of reputational 
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ratings with the scholarly productivity of faculty, Cox and Catt 
(1977)- presented data that seemed to show very little relationship 
between the Rooee and Andersen (1970) ratings of psychology depart- 
ments and the productivity of their faculty as measured- by the 
average number of articles after adjusting for the size of the 
department's faculty In 13 APA journals over a six-year period. 
Robey (1979) obtained very similar results in a study of -six political 
science journals. The Cox and Catt study would seem to call Into 
question the assumed relationship between scholarly productivity and 
ratings. However, their study was criticized on the grounds that 
they had: (1) mlsldentlfled faculty members In specific departments 
thus making their productivity measures erroneous (Klelnmuntz, 
1978); (2) Included many non-psychology faculty In the counts for 
some departments, thus making large multluniversltles appear more 
productive (Levin et al., 1978) ; (3) used the wrong number of 
faculty for departments, which makes departments that have recently 
expanded look less productive (Ross, 1978); and (4) neglected the 
fact that many psychology faculty publish in other APA journals and 
non-APA journals, and many concentrate on books (D'Amato, 1978). A 
iifth potential criticism is that some prestigious departments lay 
their claim to eminence. on the substantial contributions of their 
faculty over their entire careers rather than their recent journal 
output, and a sixth is that the ratings used were not contemporary 
with the publications data, thus introducing other possible errors. 
Finally, and perhaps most important, the Cox and Catt study, as well as 
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virtually all others, did no* Include any measures of the educational 
quality of the departments as places to study or to work. 

The purpose of this paper is to examine the question of the 
relationship between reputational ratings and productivity and 
the relationship of these to other indicators of the educational 
quality of departments by analyzing a set of data, from a national 
project designed to assess the dimensions of quality in graduate 
education (Clark, Hartnett, & Baird, 1976). These data meet the 
seven criticisms just outlined by: (1) obtaining data about produc- 
tivity directly from faculty members, thus eliminating .^identifica- 
tion; (2) including only bona fide faculty members of departments; 
(3) using only the N's of respondents, thus making the base N's 
for averages more accurate (the response rate was high); (4) including 
all journal publications, not just- those in a specific set of 
'journals, and also obtaining data on book and monograph publications; 
(5) obtaining publication figures for each faculty member's entire 
career, as well as the most recent period; (6) obtaining ratings 
contemporaneously with the publication data; and (7) including a 
variety of measures of the quality of the department's environment 
for learning and working. In addition,- the data comes from three 
fields, chemistry, history, and psychology so that the patterns 
can be compared across fields. 



Method 



A . p„r„l,-v Sample* «nd Variables . The" basic sample consists of 
511 chemistry, 584 history, and 598 psychology faculty members in 



a national sample of doctoral programs. There were 24 chemistry 
departments, 25 history departments, and 24 psychology departments. 
The departments were selected to constitute a heterogenous sample 
in terms of size, location, earlier reputational ratings, and 
emphasis. <Gee Clark, Hartnett, & Baird, 1976, for details.) 

Questionnaires concerning faculty activities and views were * 
developed by the authors and distributed by departmental chairpersons. 
(The average response rate was 75 percent). Part of the questionnaire 
asked faculty to report the number of their: (1) professional 
articles in journals and chapters in books, (2) scholarly book 
reviews, and (3) books and monographs. This information was requested 
separately for the entire career and the last three years. The 
validity of this self-report information is suggested by an analysis 
of the data by Clark and Centra (forthcoming) showing that the 
self-reports of the last three year journal production by young 
psychologists correlated .85 with counts based on listings in 
Psychological Abstracts , which, pf course, does not cover non- 
psychology journals. 

The following six measures were used to assess productivity: 

1. Articles and book chapters, total for the entire career. 

2. Articles, book chapters and book reviews, total for the entire 
career. 

3. Books and monographs, total for entire career. 

4. Articles and book chapters, last three years. 

5. Articles, book chapters and book reviews, last three years. 

6. Books and monographs, last three years. 
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Faculty were asked to rate the quality of the faculty in the 
other departments In their field In the study using exactly the same 
procedure used by Roose and Andersen (1970). 

Other variables based on faculty responses Included: 

Compatibility of Work Environment Scale , a six-Item scale with 
a coefficient alpha of .77 designed to measure the degree of 
stress and the level of morale of faculty. 

Ratine of the Quality of Facultv-Student Relations, the mean 
departmental rating based on faculty ratings on a four-point scale. 

Ratine of the Humaneness of the Environment, the mean departmental 
rating based on faculty ratings on a four-point scale. 

In addition to the faculty survey, graduate students and alumni 
from the program were surveyed. 

B . Student sacpie* and variables . Returned questionnaires were 
obtained from 791 chemistry, 893 history, and 967 psychology 
graduate students. The variables based on their responses that are 
reported in this paper include: 

Quality of Teachine scale , a suven-item scale with a coefficient 
alpha of .83, designed to measure students' perceptions of a variety 
of faculty behaviors associated with good teaching. 

Quality of the Environme nt for Learning Scale, a six-item scale 
with a coefficient alpha of .73, designed to assess the supportiveness 
and fairness of the department's environment. 

Faulty Concern for Students Scale , a seven-item scale with * 
coefficient alpha of .80 designed to measure students' perceptions 
of the faculty's accessibility, interest, and helpfulness. 
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Satisf action t-h» Program Scale, a four-item scale with 

a coefficient alpha of .87 designed to assess overall satisfaction 
and evaluation of the program. 

Quality of the Curriculum Scale , a seven-item scale with a 
coefficient alpha of .80, designed to assess students' evaluations 
of the flexibility, depth, and breadth of the curriculum. 

Relevance of Degree Requirement, Scale , a seven-item scale \ 
with a coefficient alpha of .72, designed to assess students' 
evaluations of the appropriateness and relevance of various degree 
requirements • 

Strength With Which Stude nt Would Advise a Friend to Come 
to the Department , a rating made on a four-point scale. 
,C. Alumni data . Questionnaires from 430 chemistry, 349 history, 
and 393 psychology doctoral recipients were returned. The variables 
based on their responses include: 

Satisf action With Disserts -*™ Kvnerience Scale, an eleven-item 
scale with a coefficient alpha of .85 designed to measure alumni 
satisfaction with a variety of aspects of the dissertation experience. 

V»lue of the Program fo^ PrPBPnt- Work Rating, the mean rating 
across seven aspects of the doctoral program. 

Average Annual Number' of A rticles and Book Reviews, a self-report 
measure of alumni scholarly productivity. 

Strength' With Which Alui m H Would Advise Friends to Go to the 
Department , a rating made on a four-point scale. 
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Aialvses . Departments were grouped into three "quality" groups 
based on their reputational ratings: high, medium, and low. The 
publications o£ faculty in each group as measured in the three areas 
listed above, for the entire career and for the last three years, 
was calculated and one-way analysis of variance across the 
groups performed. Distribution analyses were also made. The mean 
reputational rating of each department and the mean number of publi- 
cations on each index was also calculated, and these means correlated 
to further examine the relationship. Each of these means was also 
correlated with the means on the other variables just described. 

Results 

As shown in Table 1, the analysis of variance results showed 
that all four publications measures involving articles, book 
chapters, and reviews differed significantly across the three 
reputational ratings groups for all three fields. (The measures 
of article and book chapter production are designed to assess 
research productivity: the measures including book reviews are 
designed to assess overall scholarly productivity). The trends are 
most clear for the measures covering the entire career, where the 
high rated departments produced approximately twice as much per 
faculty member as the low rated departments, but are much less 
clear for the measures over the last three years. In fact, for 
history, the least productive group on measure three were faculty 
departments rated in the middle rather 'than those rated low. In 
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Table 1 

Productivity of Faculty in Departments Rated Low, Medium, 
and High in "Quality": Anova Results 

Means . _ 

Total Rating Group 



Group Low Medium High 



1. 


Articles and 
Book Chapters: 
Entire Career - 


C 
H 
P 


54.2 
11.9 
27.1 


29.2 
9.5 
17.1 


51.6 
10.7 
25.2 


71.5 
14.7 
32.6 


17.9 
6.6 
12.6 


<;001 
.002 
<.001 


2. 


Articles, Book C 
Chapters & Reviews : H 
Entire Career P 


56.8 
29.4 
29.8 


30,8 
21.8 
18.5 


54.4 
27.6 
27.8 


74.3 
35.3 
35.6 


17.5 
6.2 
12.4 


<.001 
.002 
<.001 


3. 


Books and 
Monographs : 
Entire Career 


C 
K 
P 


1.5 
3.4 
2.3 


1.3 
3.2 
1.3 


1,5 
3.1 
1.8 


1.7 
3.9 
3.3 


.3 
' 2.1 
10.4 


. NS 
NS 
<.001 


4. 


Articles and 
Book Chapters: 
Last Three Years 


C 
H 
P 


11.2 
3.0 
6.7 


5.7 
3.0 
5.1 


10.5 
2.6 
6.6 


15.2 

*.4 
/.3 


24.9 
4.3 
4.8 


<.001 
.014 
.009 


5. 


Articles, Book 
Chapters & Reviews 
Last Three Years 


C 
:H 
P 


11.8 
7.0 
7*3 


6.1 
5.9 
5.5 


11.3 , 
6.6 
7.4 


15.6 
8.0 
7.8 


24.0 
4.3 
4.6 


<.001 
.014 
.011 


6. 


-Books and 
Monograph: 
Last Thi-e Years 


C 
H 
P 


.5 
1.0 
.8 


.5' 
1.1 
.5 


.5 
.9 
.8 


. .6 
1.1 
1.0 


.1 
1.9 
3.4 


NS 
NS 
.032 



C - Chemistry 
H - History 
P - Psychology 



addition, t-tests showed that the differences between psychology 
departments rated in the middle and departments rated high were 
not significant for the last three^asures. 

- The measures of book and monograph production were not signifi- 
cantly different across faculty in departments with different 
ratings for chemistry and history, m addition, as Just noted, the 
differences between publication rates of psychology faculty in depart- 
meats rated high and medium on measure six were not significantly 
different, inns, book production does not appear to be particularly 

related to reputational ratings. 

To examine the possibility that the high average publication 
levels of the h*gh rated departments were due to- a few very-produc- 
tive faculty, an analysis of the distribution of faculty publication 
rates ih each group was made. There was no sign of concentrations 
of extraordinarily higti producers: the higher productivity of the 
high rated departments was uniform across the distributions. 

The correlations of departmental mean publication Indices with 
the mean reputational ratings and with each other are shown In Table 2. 
The level of these correlations generally suggests a moderate 
relationship between reputation and productivity. The only exception 
wae that article and book review production was highly correlated 
with ratios in chemistry. The correlations also suggest that 
publication rates per faculty member of Journal articles and books 
are typically only moderately related, if at all. 



Table 2 

Intercorrelations of Mean Publication Rates and Reputational Ratings 





(1) 


(2) 


(3) 


(4) 


(5) 




Chemistry 










(1) Articles, Chapters and Book 
Reviews : Career 


1.00 










(2) Articles, Chapters and Book Reviews: 

Last Three Years 1 « 59 


1.00 


1.00 






(3) Books, Monographs: Career 


.34 


.10 




• 


(4) Books, Monographs: 

Last Three Years 

(5) Reputational Ratings 


.23 
.85** 

History 


.18 
.90** 


.74* 
.34 


1.00 
.19 


1.00 


(1) Articles, Chapters and Book 
Reviews: Career 


1.00 










(2) Articles, Chapters and Book Reviews: 

Last Three Years 

(3) Books, Monographs: Career .66** 


1.00 
.21 


1.00 






(4) Books, Monographs: 

Last Three Years 

(5) Reputational Ratings 


.24 
.55** 

Psychology 


.23 
.50* 


.52** 
.52** 


1.00 
.37 


1.00 


(1) Articles, Chapters and Book 
Reviews: Career 


1.00 










(2) Articles, Chapters and Book Reviews: 

Last Three Years - 50 ** 

(3) Books, Monographs: Career .43* 


1.00 
•15 


1.00* 







(4) Books, Monographs: 

Last Three Years 

(5) Reputational Ratings 



.34 
.33 



.26 .70** 1.00 
.47* .57** .40* 



1.00 



*p < .05 
**p < .01 
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It 18 important to remember that correlations can be misleading, 
especially when the goal Is to evaluate Individual departments. For 
this reason scatterplots between mean departmental publication rates 
per faculty member and reputatlonal ratings were produced. An 
example for. psychology is given In Figure 1. Here, with departments 
shown by a code letter, it is clear that the overall correlation is 
misleading. Department I is the most productive, but is rated lower 
than Department K, the third least productive. Departments M and H 
were approximately equally productive, but M obtained the second 
highest rating, and H the third lowest. Another example, in this 
case for history, is shown in Figure 2. The figure plots peer 
ratings of history faculty horizontally and average annual publi- 
cations vertically, and again, individual programs are represented 
by -randomly assigned letters. Lines have been drawn to indicate the 
mean of each variable. Although the correlation in Figure 2 is 
fairly high (.55), the locations of Departments J, C, K, X, and P 
provide a good illustration of the way in which correlations can be 
misleading for individual program review. The faculty of Department 
J, the most productive of all, has one of the lowest ratings, and *, 
the faculty of Department C, which has a rating just below that of 
Department K, averaged nearly twice as many articles per year as K. 
Department X and P, are very similar in productivity, but P is the 
highest rated program and X is rated below the mean. 

In order to better understand the meaning of ratings and 
publication indices the mean scores for the variables described in 
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Peer Ratings of Quality of the Faculty 

Figure 1. Scatterplot of Peer Ratings of Faculty Quality and Journal Publications in the 
Last Three Tears in Paychology 



18 



MM. S.O. 



0.431 



COKKfcl • ' 0.544? 





1.10 


.1 




.3.05 • 


•1 




3.01 


•1 




2.W 


..1 




2.«2 


•1 


j". ■ ■ 


M» 


•1 


/J; 


MI 


.1 




Ml 


•1 


1 


i.n 


•1 


5 


I.M 


..1 


8 


2.*S 


•1 


2.40 


•1 




1.5* 


.» 


V 


l.Sl 


•1 


1.41 


•1 


i 


1.42 


..1 




Ml 


•1 




1.11 


•1 


% 


.1.2* 


.1 • 




1.14 


•1 


M 

0 


1.20 


•1 


1.15 


..1 


1.11 


•1 


o 


2.3* 


If" 


2.02 


•1 




1.11 


•1 


s 




• | 




i.m 


..1 


1 


1.14 


•1 


«p - 


t.M 


•1 




LIS 


.1 


t.TI 


•1 


S 


I.M 


•1 


1 


1.12 


..1 




.' I.ST 


•1 


< 


l.»l 


•1 




1.41 


.1 


» 

s 


1.44 


•1 


.1.11 


• 1 


1 


'l.M 


..1 


1 


1.10 


•1 


< 


1.2* 


•1 




1.21 


.! 




1.17 


•1 


H 
3 

U 


1.12 
1.01 




& 


1.01 


•i 




o.Of 


•i 




O.H 


.i 




O.fO 


•i 




X 

ft 

§ 



2.7)00 1 
2.5000' 3*0000 



,HM,I«M 

.2500 3 
3.5000 



.7500 4. 
4.0000 



..•«,.... 

7500 4 
4.5000 



.7500 
5 



0000 



............ 

0.2500 0.7500 1.2500 M506 1.— 5 1 
0.0 0.5000 1.0000 U5000 2.0000 

Peer Ratings of Quality of the Faculty 

Figure 2. Scatterplot of Peer Ratings of Faculty Quality. and Faculty Average Annual Journal 
Publications in History 
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with the mean reputational rating of the department and the mean 
number of Journal articles published by the faculty of each depart- 
ment In the last three years. The results are shown in Table 3. * 

in chemistry ratings were unrelated to all the other indicators 
of educational quality save one. In history they were negatively 
related to two and positively related to two. Finally, in psychology 
ratings were positively related to eight indicators. In contrast, 
publications were positively related to three Indicators In chemistry 
none In history, and one in psychology,, There were no consistent 
patterns across fields; none of the other indicators were significantly 
related to ratings or publications across all three fields, and only 
tW o-the faculty Compatibility of Work En- ronmant Scale and the 
alumni's recommendation to friends to come to the department-were 
related in two field's, both to ratings. 

Although these results are quite complex, there appear to be 
several trends. First, student Indicators of the quality of human 
relationships are negatively related to ratings in history. (And 
the positive relationship of ratings to alumni's recommendation 
of the department to friends is counterbalanced by a near zero 
relation to a similar recommendation by currently enrolled students). 
Second, several other student indicators of the quality of the 

"overTira ' 

psychology. Third, average publication rates did not seem to-be- 
especially related to most of the measures. Although it is tempting 
to speculate on the reasons for these trends, and the differences 
among fields, it seems clear that/overall, ratings are more strongly 

7 20 



"# 



Table 3 

Correlations of Reputational Ratings and Publications with 
Other Indicators of Educational Quality 



Reputational Ratings 
C H P 



Journal Publications 
Last Three Years 



C 



H 



P 



From Graduate Student Data 

1, Quality of teaching scale 

2, Quality of environment for learning scale 

3, DegTMi of faculty concern for students scale 
, 4, Satisfaction 14th program scale 

5, Quality of the curriculum scale - 

6. Relevance of degree requirements scale 
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related to student and alumni descriptions and evaluations of the 
programs than are faculty publication rates In psychology, but that 
neither measure Is consistently related to other Indicators of 
educational quality In chemistry and history. 

Discussion 

These analyses showed statistically significant relationships 
between departmental reputatlonal ratings and a variety of publication 
measures. However, the results also showed that the ratings tend to 
reflect the career publications of the faculty more than their 
recent publications. The relationship as shown in the correlations 
was also far from perfect, except in chemistry. And, finally, as 
shown in the scatterplots, the ratings do not reflect the productivity 
of individual departments at all wall. Thus, the utility of the 
ratings as an Indicator of the scholarly contribution of single 
departments is questionable. 

These results also indicate that any particular publicatipn 
measure should not be used as a measure of overall departmental 
"quality. M The correlation results showed that the different 
measures of productivity were imperfectly related; some departments 
that were high on b6ok production were low on journal production. 
Using only book or journal production would present a taise picture 
of the contributions of individual department^. (The use of publication 
measures becomes even more problematical when the "quality" of the 
publications is taken into account) . However, more important are 
the analyses that show both reputatlonal ratings and publication 



figures typically to be unrelated to the quality of teaching as 
reported by students, alumni ratings of the quality of their prepara- 
tion, faiulty concern for students, etc., at least in departments of 

■ ' / ' 
chemistry and history. 

Mutational ratings were related to some of these measures in 
psychology, but a measure of faculty publication was not. This 
result: suggests that faculty productivity of articles and books may 
be an adequate measure of one aspect of a department's contribution 
to the field of psychology, but it is only one aspect of many, 
further, although ratings were significantly related to some indica- 
tors of the educational excellence of the department, they were not 
related to others, particularly those reflecting the quality of 1 
student-faculty relationships and the overall environment for 
learning, which suggests that these variables can vary with consider- 
able independence. It has been argued elsewhere (e.g., Hartnett, 
Clark, TMJaird, 1978) that what are needed are multiple indicators 
of the multiple aspects of departmental educational and scholarly 
excellence. This would allow one to construct a "profile" of 
departmental strengths and weaknesses (Clark, 1980) which seems 
fairer and more sound than attempting to evaluate a department's 
contribution with one or two indicators. This is especially true in 
the current period when the amount of financial support for depart- 
ments is under careful scrutiny and, in some cases, when the continued 
existence of departments is being questioned. 
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